---

title: Noisy student


keywords: fastai
sidebar: home_sidebar

summary: "Callback to apply noisy student self-training (a semi-supervised learning approach) based on: Xie, Q., Luong, M. T., Hovy, E., & Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10687-10698)."
description: "Callback to apply noisy student self-training (a semi-supervised learning approach) based on: Xie, Q., Luong, M. T., Hovy, E., & Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10687-10698)."
nb_path: "nbs/061_callback.noisy_student.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/061_callback.noisy_student.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="NoisyStudent" class="doc_header"><code>class</code> <code>NoisyStudent</code><a href="https://github.com/timeseriesAI/tsai/tree/main/tsai/callback/noisy_student.py#L25" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>NoisyStudent</code>(<strong><code>dl2</code></strong>:<code>DataLoader</code>, <strong><code>bs</code></strong>:<code>Optional</code>[<code>int</code>]=<em><code>None</code></em>, <strong><code>l2pl_ratio</code></strong>:<code>int</code>=<em><code>1</code></em>, <strong><code>batch_tfms</code></strong>:<code>Optional</code>[<code>list</code>]=<em><code>None</code></em>, <strong><code>do_setup</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>pseudolabel_sample_weight</code></strong>:<code>float</code>=<em><code>1.0</code></em>, <strong><code>verbose</code></strong>=<em><code>False</code></em>) :: <code>Callback</code></p>
</blockquote>
<p>A callback to implement the Noisy Student approach. In the original paper this was used in combination with noise:</p>

<pre><code>- stochastic depth: .8
- RandAugment: N=2, M=27
- dropout: .5

</code></pre>
<p>Steps:</p>

<pre><code>1. Build the dl you will use as a teacher
2. Create dl2 with the pseudolabels (either soft or hard preds)
3. Pass any required batch_tfms to the callback</code></pre>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">tsai.data.all</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">from</span> <span class="nn">tsai.models.all</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">from</span> <span class="nn">tsai.tslearner</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">dsid</span> <span class="o">=</span> <span class="s1">&#39;NATOPS&#39;</span>
<span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">splits</span> <span class="o">=</span> <span class="n">get_UCR_data</span><span class="p">(</span><span class="n">dsid</span><span class="p">,</span> <span class="n">return_split</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">pseudolabeled_data</span> <span class="o">=</span> <span class="n">X</span>
<span class="n">soft_preds</span> <span class="o">=</span> <span class="kc">True</span>

<span class="n">pseudolabels</span> <span class="o">=</span> <span class="n">ToNumpyCategory</span><span class="p">()(</span><span class="n">y</span><span class="p">)</span> <span class="k">if</span> <span class="n">soft_preds</span> <span class="k">else</span> <span class="n">OneHot</span><span class="p">()(</span><span class="n">y</span><span class="p">)</span>
<span class="n">dsets2</span> <span class="o">=</span> <span class="n">TSDatasets</span><span class="p">(</span><span class="n">pseudolabeled_data</span><span class="p">,</span> <span class="n">pseudolabels</span><span class="p">)</span>
<span class="n">dl2</span> <span class="o">=</span> <span class="n">TSDataLoader</span><span class="p">(</span><span class="n">dsets2</span><span class="p">,</span> <span class="n">num_workers</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">noisy_student_cb</span> <span class="o">=</span> <span class="n">NoisyStudent</span><span class="p">(</span><span class="n">dl2</span><span class="p">,</span> <span class="n">bs</span><span class="o">=</span><span class="mi">256</span><span class="p">,</span> <span class="n">l2pl_ratio</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">learn</span> <span class="o">=</span> <span class="n">TSClassifier</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">splits</span><span class="o">=</span><span class="n">splits</span><span class="p">,</span> <span class="n">batch_tfms</span><span class="o">=</span><span class="p">[</span><span class="n">TSStandardize</span><span class="p">(),</span> <span class="n">TSRandomSize</span><span class="p">(</span><span class="mf">.5</span><span class="p">)],</span> <span class="n">cbs</span><span class="o">=</span><span class="n">noisy_student_cb</span><span class="p">)</span>
<span class="n">learn</span><span class="o">.</span><span class="n">fit_one_cycle</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>labels / pseudolabels per training batch              : 171 / 85
relative labeled/ pseudolabel sample weight in dataset: 4.0
</pre>
</div>
</div>

<div class="output_area">


<div class="output_html rendered_html output_subarea ">
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>epoch</th>
      <th>train_loss</th>
      <th>valid_loss</th>
      <th>accuracy</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>1.817813</td>
      <td>1.812490</td>
      <td>0.044444</td>
      <td>00:06</td>
    </tr>
  </tbody>
</table>
</div>

</div>

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>
X: torch.Size([171, 24, 51])  X2: torch.Size([85, 24, 51])  X_comb: torch.Size([256, 24, 37])
y: torch.Size([171])  y2: torch.Size([85])  y_comb: torch.Size([256])
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">pseudolabeled_data</span> <span class="o">=</span> <span class="n">X</span>
<span class="n">soft_preds</span> <span class="o">=</span> <span class="kc">False</span>

<span class="n">pseudolabels</span> <span class="o">=</span> <span class="n">ToNumpyCategory</span><span class="p">()(</span><span class="n">y</span><span class="p">)</span> <span class="k">if</span> <span class="n">soft_preds</span> <span class="k">else</span> <span class="n">OneHot</span><span class="p">()(</span><span class="n">y</span><span class="p">)</span>
<span class="n">dsets2</span> <span class="o">=</span> <span class="n">TSDatasets</span><span class="p">(</span><span class="n">pseudolabeled_data</span><span class="p">,</span> <span class="n">pseudolabels</span><span class="p">)</span>
<span class="n">dl2</span> <span class="o">=</span> <span class="n">TSDataLoader</span><span class="p">(</span><span class="n">dsets2</span><span class="p">,</span> <span class="n">num_workers</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">noisy_student_cb</span> <span class="o">=</span> <span class="n">NoisyStudent</span><span class="p">(</span><span class="n">dl2</span><span class="p">,</span> <span class="n">bs</span><span class="o">=</span><span class="mi">256</span><span class="p">,</span> <span class="n">l2pl_ratio</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">learn</span> <span class="o">=</span> <span class="n">TSClassifier</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">splits</span><span class="o">=</span><span class="n">splits</span><span class="p">,</span> <span class="n">batch_tfms</span><span class="o">=</span><span class="p">[</span><span class="n">TSStandardize</span><span class="p">(),</span> <span class="n">TSRandomSize</span><span class="p">(</span><span class="mf">.5</span><span class="p">)],</span> <span class="n">cbs</span><span class="o">=</span><span class="n">noisy_student_cb</span><span class="p">)</span>
<span class="n">learn</span><span class="o">.</span><span class="n">fit_one_cycle</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>labels / pseudolabels per training batch              : 171 / 85
relative labeled/ pseudolabel sample weight in dataset: 4.0
</pre>
</div>
</div>

<div class="output_area">


<div class="output_html rendered_html output_subarea ">
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>epoch</th>
      <th>train_loss</th>
      <th>valid_loss</th>
      <th>accuracy</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>1.827661</td>
      <td>1.820120</td>
      <td>0.088889</td>
      <td>00:10</td>
    </tr>
  </tbody>
</table>
</div>

</div>

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>
X: torch.Size([171, 24, 51])  X2: torch.Size([85, 24, 51])  X_comb: torch.Size([256, 24, 67])
y: torch.Size([171, 6])  y2: torch.Size([85, 6])  y_comb: torch.Size([256, 6])
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

</div>
 

